We address the problem of meta-learning which learns a prior over hypothesis from a sample of meta-training tasks for fast adaptation on meta-testing tasks. A particularly simple yet successful paradigm for this research is model-agnostic meta-learning (MAML). Implementation and analysis of MAML, however, can be tricky; first-order approximation is usually adopted to avoid directly computing Hessian matrix but as a result the convergence and generalization guarantees remain largely mysterious for MAML. To remedy this deficiency, in this paper we propose a minibatch proximal update based meta-learning approach for learning to efficient hypothesis transfer. The principle is to learn a prior hypothesis shared across tasks such that the minibatch risk minimization biased regularized by this prior can quickly converge to the optimal hypothesis in each training task. The prior hypothesis training model can be efficiently optimized via SGD with provable convergence guarantees for both convex and non-convex problems. Moreover, we theoretically justify the benefit of the learnt prior hypothesis for fast adaptation to new few-shot learning tasks via minibatch proximal update. Experimental results on several few-shot regression and classification tasks demonstrate the advantages of our method over state-of-the-arts.

Neural Information Processing Systems http://nips.cc/

Hessian matrix but as a result the convergence and generalization guarantees remain largely mysterious for MAML.

UPDATE: I'd like to thank the authors for their detailed response. In light of this response I have increased my score to a 7. Originality This paper presents Meta-MinibatchProx, an algorithm for model and algorithm agnostic meta learning that, unlike MAML and friends, comes with theoretical guarantees of convergence. To the best of my knowledge it is the first such algorithm to offer any convergence guarantees, and also has the potential to scale to very large problems. Quality Something I would like to have seen addressed explicitly in this paper is the distinction between what I will call the "finite" and "infinite" versions of MMP. The distinction here is related to the comment "Usually we are only provided with n observed tasks..." on line 135.

All three reviewers agree that the paper is novel, well presented and shows convincing empirical results. The extensive theoretical analysis of the convergence of their algorithm makes the paper particularly interesting and different from prior related art. I agree with Reviewer 3, and encourage the authors to add a high level explanation of the difference between MMP and MAML, similar to the one provided in the author's response.

We address the problem of meta-learning which learns a prior over hypothesis from a sample of meta-training tasks for fast adaptation on meta-testing tasks. A particularly simple yet successful paradigm for this research is model-agnostic meta-learning (MAML). Implementation and analysis of MAML, however, can be tricky; first-order approximation is usually adopted to avoid directly computing Hessian matrix but as a result the convergence and generalization guarantees remain largely mysterious for MAML. To remedy this deficiency, in this paper we propose a minibatch proximal update based meta-learning approach for learning to efficient hypothesis transfer. The principle is to learn a prior hypothesis shared across tasks such that the minibatch risk minimization biased regularized by this prior can quickly converge to the optimal hypothesis in each training task.

We address the problem of meta-learning which learns a prior over hypothesis from a sample of meta-training tasks for fast adaptation on meta-testing tasks. A particularly simple yet successful paradigm for this research is model-agnostic meta-learning (MAML). Implementation and analysis of MAML, however, can be tricky; first-order approximation is usually adopted to avoid directly computing Hessian matrix but as a result the convergence and generalization guarantees remain largely mysterious for MAML. To remedy this deficiency, in this paper we propose a minibatch proximal update based meta-learning approach for learning to efficient hypothesis transfer. The principle is to learn a prior hypothesis shared across tasks such that the minibatch risk minimization biased regularized by this prior can quickly converge to the optimal hypothesis in each training task.